Robotic system with enhanced scanning mechanism

ABSTRACT

A method for operating a robotic system including determining an initial pose of a target object based on imaging data, wherein the initial pose is for estimating a resting orientation of the target object in a pickup area; calculating a confidence measure associated with the initial pose, wherein the confidence measure is for representing a likelihood of the initial pose being correct; and calculating a motion plan according to the confidence measure, the motion plan for executing a task based on picking up the target object from a start location, transferring the target object to a task location, and scanning one or more object identifiers between the start location and the task location.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims the benefit of, U.S.patent application Ser. No. 16/258,120, filed Jan. 25, 2019, which isincorporated by reference herein in its entirety.

This application contains subject matter related to U.S. patentapplication Ser. No. 16/546,209, filed on Aug. 20, 2019, U.S. patentapplication Ser. No. 16/546,226, filed on Aug. 20, 2019, and U.S. patentapplication Ser. No. 16/546,236, filed on Aug. 20, 2019, all titled “AROBOTIC SYSTEM WITH ENHANCED SCANNING MECHANISM,” and each incorporatedherein by reference in their entireties.

TECHNICAL FIELD

The present disclosure is directed generally to robotic systems and,more specifically, to systems, processes, and techniques for scanningobjects.

BACKGROUND

With their ever-increasing performance and lowering cost, many robots(e.g., machines configured to automatically/autonomously executephysical actions) are now extensively used in many fields. Robots, forexample, can be used to execute various tasks (e.g., manipulate ortransfer an object through space) in manufacturing and/or assembly,packing and/or packaging, transport and/or shipping, etc. In executingthe tasks, the robots can replicate human actions, thereby replacing orreducing the human involvement that would otherwise be required toperform dangerous or repetitive tasks.

However, despite the technological advancements, robots often lack thesophistication necessary to duplicate human sensitivity and/oradaptability required for executing more complex tasks. For example,manipulation robots often lack the granularity of control andflexibility in the executed actions to account for deviations oruncertainties that may result from various real-world factors.Accordingly, there remains a need for improved techniques and systemsfor controlling and managing various aspects of the robots to completethe tasks despite the various real-world factors.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of an example environment in which a roboticsystem with an enhanced scanning mechanism may operate.

FIG. 2 is a block diagram illustrating the robotic system in accordancewith one or more embodiments of the present disclosure.

FIG. 3A is an illustration of an object in a first pose.

FIG. 3B is an illustration of the object of FIG. 3A in a second pose.

FIG. 3C is an illustration of the object of FIG. 3A in a third pose.

FIG. 4 is a top view illustrating an example task executed by therobotic system in accordance with one or more embodiments of the presentdisclosure.

FIG. 5A is a flow diagram for operating the robotic system of FIG. 1 inaccordance with one or more embodiments of the present disclosure.

FIG. 5B is a flow diagram for deriving motion plans based on scanningmetrics in accordance with one or more embodiments of the presentdisclosure.

DETAILED DESCRIPTION

Systems and methods for a robotic system with an enhanced scanningmechanism are described herein. The robotic system (e.g., an integratedsystem of devices that executes one or more designated tasks) configuredin accordance with some embodiments provides enhanced scanning byderiving and executing motion plans according to uncertaintiesassociated with initial poses of objects.

The robotic system can be configured to execute a task based onmanipulating (e.g., physically displacing and/or reorienting) a targetobject. For example, the robotic system can sort or relocate variousobjects based on picking the target object from a source location (e.g.,a bin, a pallet, or a conveyer belt) and moving it to a destinationlocation. In some embodiments, the task can further include scanning thetarget object during transfer, such as by presenting one or moreidentifiers (e.g., barcodes or quick response (QR) codes) located on oneor more specific locations and/or surfaces of the target object to a setof scanners. Accordingly, the robotic system can derive or calculate amotion plan to grip and pick up the target object, transfer the targetobject to a presentation location/orientation to present the identifiersto the scanners, and place the target object at a task location (e.g.,by transferring the object to the task location, adjusting the pose ofthe object, lowering the object, and/or releasing the object).

To execute the task, in some embodiments, the robotic system can includean imaging device (e.g., a camera, an infrared sensor/camera, a radar, alidar, etc.) used to identify a location and a pose (e.g., a restingorientation) of the target object and/or the environment around thetarget object. In some embodiments, the robotic system can furthercalculate a confidence measure associated with the pose. The confidencemeasure can represent a measure of certainty or likelihood that thedetermined pose matches the actual real-world pose of the target object.For illustrative example, the robotic system can obtain images (e.g.,images of a pickup area, such as a source bin or pallet) that depictlocations and orientations of objects that are tasked to be transferredfrom a pickup area to a task area (e.g., destination bin or pallet). Therobotic system can process the images to identify or select the targetobject according to a predetermined order (e.g., from top to bottomand/or from an outer edge and inward). The robotic system can furtherdetermine the initial pose from the image, such as by identifying andgrouping object lines (e.g., according to pixel color, brightness,and/or change in values thereof relative to adjacent pixels). Indetermining the initial pose, the robotic system can further calculatethe confidence measure (e.g., a quantified degree of certaintyassociated with the determined pose) according to a predeterminedprocess and/or equation.

According to the location, the pose, the confidence measure, or acombination thereof, the robotic system can derive and execute a motionplan (e.g., a sequence of controls for the actuators for moving one ormore links and/or joints) to execute the task. For example, for sortingand/or relocating the target object, the motion plan can correspond togripping the target object initially at the source location,manipulating it across space, and placing it at the destinationlocation.

Traditional systems derive and execute motion plans strictly based ondetermined poses of the object. Accordingly, the traditional systemsderive and execute motion plans regardless of any deviations, errors,and/or uncertainties that may have occurred upstream (e.g., in gatheringthe input data). As such, the traditional systems cannot mitigate orcorrect the deficiencies introduced upstream, which leads to taskfailures (e.g., failures in identifying objects and/or losing piecesduring transfer) that require human intervention/input.

Unlike the traditional systems, various embodiments described below canderive and execute the motion plan according to the confidence measure.In other words, the robotic system described below can vary an approachto the target object, change a grip location on the target object,change a presentation pose/location of the target object, and/or changeother portions of the motion path according to the confidence measure.As an illustrative example, the robotic system can select as a targetobject a box located in the pickup area. For this example, the boxcorresponds to a pose where an object-top surface is generally orientedhorizontally and exposed, and one of the object-side surfaces (i.e.,smaller/narrower than the top surface) is generally oriented verticallyand also exposed. The robotic system can include in master data that theobject has one identifier on an object-bottom surface (i.e., opposite tothe object-top surface) and a smaller identifier on one of theobject-side surfaces. When the robotic system processes the image of thepickup location in identifying the target object, the robotic system cancalculate the confidence measure. For example, the confidence measurecan correspond to a measure of the match between one or more visiblecharacteristics of the box (e.g., a shape, a color, an image, a design,a logo, a text, etc.) captured in the image to predetermined informationin the master data. If the confidence measure is above a threshold, suchthat the robotic system recognizes with sufficient certainty that theobject-top surface is exposed on top of the box, the robotic system canplace an end-effector over the exposed top surface, grip the topsurface, and rotate the target object to present a bottom surface at afixed location before a scanner. If the confidence measure is below athreshold, such that the robotic system cannot recognize whether the topsurface or the bottom surface is exposed, the robotic system can placethe end-effector by one of the object-side surfaces, grip theobject-side surface, and rotate the target object to pass between a setof opposing scanners.

Scanning the target object in the air (e.g., at a location between thestart location and the task location) provides improved efficiency andspeed for performing the task. By calculating the motion plan thatincludes the scanning locations and also coordinates with the objectscanners, the robotic system can effectively combine the task fortransferring the target object with the task for scanning the targetobject. Moreover, deriving the motion plan based on the confidencemeasure of the initial orientation further improves efficiency, speed,and accuracy for the scanning task. The robotic system can calculate themotion plan that accounts for alternative orientations that correspondto a hypothesis that the initial pose is inaccurate. Accordingly, therobotic system can increase the likelihood of accurately/successfullyscanning the target object even with pose determination errors (e.g.,resulting from calibration errors, unexpected poses, unexpected lightingconditions, etc.). The increased likelihood in accurate scans can leadto increased overall throughput for the robotic system and furtherreduce operator efforts/interventions. Details regarding the confidencecalculation and the associated path calculation are described below.

In the following description, numerous specific details are set forth toprovide a thorough understanding of the present disclosure. In otherembodiments, the techniques introduced here can be practiced withoutthese specific details. In other instances, well-known features, such asspecific functions or routines, are not described in detail in order toavoid unnecessarily obscuring the present disclosure. References in thisdescription to “an embodiment,” “one embodiment,” or the like mean thata particular feature, structure, material, or characteristic beingdescribed is included in at least one embodiment of the presentdisclosure. Thus, the appearances of such phrases in this specificationdo not necessarily all refer to the same embodiment. On the other hand,such references are not necessarily mutually exclusive either.Furthermore, the particular features, structures, materials, orcharacteristics can be combined in any suitable manner in one or moreembodiments. It is to be understood that the various embodiments shownin the figures are merely illustrative representations and are notnecessarily drawn to scale.

Several details describing structures or processes that are well-knownand often associated with robotic systems and subsystems, but that canunnecessarily obscure some significant aspects of the disclosedtechniques, are not set forth in the following description for purposesof clarity. Moreover, although the following disclosure sets forthseveral embodiments of different aspects of the present disclosure,several other embodiments can have different configurations or differentcomponents than those described in this section. Accordingly, thedisclosed techniques can have other embodiments with additional elementsor without several of the elements described below.

Many embodiments or aspects of the present disclosure described belowcan take the form of computer- or controller-executable instructions,including routines executed by a programmable computer or controller.Those skilled in the relevant art will appreciate that the disclosedtechniques can be practiced on computer or controller systems other thanthose shown and described below. The techniques described herein can beembodied in a special-purpose computer or data processor that isspecifically programmed, configured, or constructed to execute one ormore of the computer-executable instructions described below.Accordingly, the terms “computer” and “controller” as generally usedherein refer to any data processor and can include Internet appliancesand handheld devices (including palm-top computers, wearable computers,cellular or mobile phones, multi-processor systems, processor-based orprogrammable consumer electronics, network computers, mini computers,and the like). Information handled by these computers and controllerscan be presented at any suitable display medium, including a liquidcrystal display (LCD). Instructions for executing computer- orcontroller-executable tasks can be stored in or on any suitablecomputer-readable medium, including hardware, firmware, or a combinationof hardware and firmware. Instructions can be contained in any suitablememory device, including, for example, a flash drive and/or othersuitable medium.

The terms “coupled” and “connected,” along with their derivatives, canbe used herein to describe structural relationships between components.It should be understood that these terms are not intended as synonymsfor each other. Rather, in particular embodiments, “connected” can beused to indicate that two or more elements are in direct contact witheach other. Unless otherwise made apparent in the context, the term“coupled” can be used to indicate that two or more elements are ineither direct or indirect (with other intervening elements between them)contact with each other, or that the two or more elements cooperate orinteract with each other (e.g., as in a cause-and-effect relationship,such as for signal transmission/reception or for function calls), orboth.

Suitable Environments

FIG. 1 is an illustration of an example environment in which a roboticsystem 100 with an enhanced scanning mechanism may operate. The roboticsystem 100 includes one or more structures (e.g., robots) configured toexecute one or more tasks. Aspects of the enhanced scanning mechanismcan be practiced or implemented by the various structures.

For the example illustrated in FIG. 1, the robotic system 100 caninclude an unloading unit 102, a transfer unit 104 (e.g., a palletizingrobot and/or a piece-picker robot), a transport unit 106, a loading unit108, or a combination thereof in a warehouse or a distribution/shippinghub. Each of the units in the robotic system 100 can be configured toexecute one or more tasks. The tasks can be combined in sequence toperform an operation that achieves a goal, such as to unload objectsfrom a truck or a van for storage in a warehouse or to unload objectsfrom storage locations and load them onto a truck or a van for shipping.For another example, the task can include moving objects from onecontainer to another. Each of the units can be configured to execute asequence of actions (e.g., operating one or more components therein) toexecute a task.

In some embodiments, the task can include manipulation (e.g., movingand/or reorienting) of a target object 112 (e.g., a box, a case, a cage,a pallet, etc. targeted for manipulation) from a start location 114 to atask location 116. For example, the unloading unit 102 (e.g., adevanning robot) can be configured to transfer the target object 112from a location in a carrier (e.g., a truck) to a location on a conveyorbelt. Also, the transfer unit 104 can be configured to transfer thetarget object 112 from one location (e.g., the conveyor belt, a pallet,or a bin) to another location (e.g., a pallet, a bin, or a cage on thetransport unit 106). For another example, the transfer unit 104 (e.g., apiece-picking robot) can be configured to transfer the target object 112from one container to another. In completing the operation, thetransport unit 106 can transfer the target object 112 from an areaassociated with the transfer unit 104 to an area associated with theloading unit 108, and the loading unit 108 can transfer the targetobject 112 (by, e.g., moving the pallet carrying the target object 112)from the transfer unit 104 to a storage location (e.g., a location onthe shelves). Details regarding the task and the associated actions aredescribed below.

For illustrative purposes, the robotic system 100 is described in thecontext of a shipping center; however, it is understood that the roboticsystem 100 can be configured to execute tasks in other environments/forother purposes, such as for manufacturing, assembly, packaging,healthcare, and/or other types of automation. It is also understood thatthe robotic system 100 can include other units, such as manipulators,service robots, modular robots, etc., not shown in FIG. 1. For example,in some embodiments, the robotic system 100 can include a depalletizingunit for transferring the objects from cage carts or pallets ontoconveyors or other pallets, a container-switching unit for transferringthe objects from one container to another, a packaging unit for wrappingthe objects, a sorting unit for grouping objects according to one ormore characteristics thereof, a piece-picking unit for manipulating(e.g., for sorting, grouping, and/or transferring) the objectsdifferently according to one or more characteristics thereof, or acombination thereof.

Suitable System

FIG. 2 is a block diagram illustrating the robotic system 100 inaccordance with one or more embodiments of the present disclosure. Insome embodiments, for example, the robotic system 100 (e.g., at one ormore of the units and/or robots described above) can includeelectronic/electrical devices, such as one or more processors 202, oneor more storage devices 204, one or more communication devices 206, oneor more input-output devices 208, one or more actuation devices 212, oneor more transport motors 214, one or more sensors 216, or a combinationthereof. The various devices can be coupled to each other via wireconnections and/or wireless connections. For example, the robotic system100 can include a bus, such as a system bus, a Peripheral ComponentInterconnect (PCI) bus or PCI-Express bus, a HyperTransport or industrystandard architecture (ISA) bus, a small computer system interface(SCSI) bus, a universal serial bus (USB), an IIC (I2C) bus, or anInstitute of Electrical and Electronics Engineers (IEEE) standard 1394bus (also referred to as “Firewire”). Also, for example, the roboticsystem 100 can include bridges, adapters, controllers, or othersignal-related devices for providing the wire connections between thedevices. The wireless connections can be based on, for example, cellularcommunication protocols (e.g., 3G, 4G, LTE, 5G, etc.), wireless localarea network (LAN) protocols (e.g., wireless fidelity (WIFI)),peer-to-peer or device-to-device communication protocols (e.g.,Bluetooth, Near-Field communication (NFC), etc.), Internet of Things(IoT) protocols (e.g., NB-IoT, LTE-M, etc.), and/or other wirelesscommunication protocols.

The processors 202 can include data processors (e.g., central processingunits (CPUs), special-purpose computers, and/or onboard servers)configured to execute instructions (e.g. software instructions) storedon the storage devices 204 (e.g., computer memory). The processors 202can implement the program instructions to control/interface with otherdevices, thereby causing the robotic system 100 to execute actions,tasks, and/or operations.

The storage devices 204 can include non-transitory computer-readablemediums having stored thereon program instructions (e.g., software).Some examples of the storage devices 204 can include volatile memory(e.g., cache and/or random-access memory (RAM)) and/or non-volatilememory (e.g., flash memory and/or magnetic disk drives). Other examplesof the storage devices 204 can include portable memory drives and/orcloud storage devices.

In some embodiments, the storage devices 204 can be used to furtherstore and provide access to processing results and/or predetermineddata/thresholds. For example, the storage devices 204 can store masterdata 252 that includes descriptions of objects (e.g., boxes, cases,and/or products) that may be manipulated by the robotic system 100. Inone or more embodiments, the master data 252 can include a dimension, ashape (e.g., templates for potential poses and/or computer-generatedmodels for recognizing the object in different poses), a color scheme,an image, identification information (e.g., bar codes, quick response(QR) codes, logos, etc., and/or expected locations thereof), an expectedweight, or a combination thereof for the objects expected to bemanipulated by the robotic system 100. In some embodiments, the masterdata 252 can include manipulation-related information regarding theobjects, such as a center-of-mass location on each of the objects,expected sensor measurements (e.g., for force, torque, pressure, and/orcontact measurements) corresponding to one or more actions/maneuvers, ora combination thereof. Also, for example, the storage devices 204 canstore object tracking data 254. In some embodiments, the object trackingdata 254 can include a log of scanned or manipulated objects. In someembodiments, the object tracking data 254 can include imaging data(e.g., a picture, point cloud, live video feed, etc.) of the objects atone or more locations (e.g., designated pickup or drop locations and/orconveyor belts). In some embodiments, the object tracking data 254 caninclude locations and/or orientations of the objects at the one or morelocations.

The communication devices 206 can include circuits configured tocommunicate with external or remote devices via a network. For example,the communication devices 206 can include receivers, transmitters,modulators/demodulators (modems), signal detectors, signalencoders/decoders, connector ports, network cards, etc. Thecommunication devices 206 can be configured to send, receive, and/orprocess electrical signals according to one or more communicationprotocols (e.g., the Internet Protocol (IP), wireless communicationprotocols, etc.). In some embodiments, the robotic system 100 can usethe communication devices 206 to exchange information between units ofthe robotic system 100 and/or exchange information (e.g., for reporting,data gathering, analyzing, and/or troubleshooting purposes) with systemsor devices external to the robotic system 100.

The input-output devices 208 can include user interface devicesconfigured to communicate information to and/or receive information fromhuman operators. For example, the input-output devices 208 can include adisplay 210 and/or other output devices (e.g., a speaker, a hapticscircuit, or a tactile feedback device, etc.) for communicatinginformation to the human operator. Also, the input-output devices 208can include control or receiving devices, such as a keyboard, a mouse, atouchscreen, a microphone, a user interface (UI) sensor (e.g., a camerafor receiving motion commands), a wearable input device, etc. In someembodiments, the robotic system 100 can use the input-output devices 208to interact with the human operators in executing an action, a task, anoperation, or a combination thereof.

The robotic system 100 can include physical or structural members (e.g.,robotic manipulator arms) that are connected at joints for motion (e.g.,rotational and/or translational displacements). The structural membersand the joints can form a kinetic chain configured to manipulate anend-effector (e.g., the gripper) configured to execute one or more tasks(e.g., gripping, spinning, welding, etc.) depending on the use/operationof the robotic system 100. The robotic system 100 can include theactuation devices 212 (e.g., motors, actuators, wires, artificialmuscles, electroactive polymers, etc.) configured to drive or manipulate(e.g., displace and/or reorient) the structural members about or at acorresponding joint. In some embodiments, the robotic system 100 caninclude the transport motors 214 configured to transport thecorresponding units/chassis from place to place.

The robotic system 100 can include the sensors 216 configured to obtaininformation used to implement the tasks, such as for manipulating thestructural members and/or for transporting the robotic units. Thesensors 216 can include devices configured to detect or measure one ormore physical properties of the robotic system 100 (e.g., a state, acondition, and/or a location of one or more structural members/jointsthereof) and/or of a surrounding environment. Some examples of thesensors 216 can include accelerometers, gyroscopes, force sensors,strain gauges, tactile sensors, torque sensors, position encoders, etc.

In some embodiments, for example, the sensors 216 can include one ormore imaging devices 222 (e.g., visual and/or infrared cameras,2-dimensional and/or 3-dimensional imaging cameras, distance measuringdevices such as lidars or radars, etc.) configured to detect thesurrounding environment. The imaging devices 222 can generaterepresentations of the detected environment, such as digital imagesand/or point clouds, used for implementing machine/computer vision(e.g., for automatic inspection, robot guidance, or other roboticapplications). As described in further detail below, the robotic system100 (via, e.g., the processors 202) can process the digital image and/orthe point cloud to identify the target object 112 of FIG. 1, the startlocation 114 of FIG. 1, the task location 116 of FIG. 1, a pose of thetarget object 112, a confidence measure regarding the start location 114and/or the pose, or a combination thereof.

For manipulating the target object 112, the robotic system 100 (e.g.,via the various units) can capture and analyze an image of a designatedarea (e.g., a pickup location, such as inside the truck or on theconveyor belt) to identify the target object 112 and the start location114 thereof. Similarly, the robotic system 100 can capture and analyzean image of another designated area (e.g., a drop location for placingobjects on the conveyor belt, a location for placing objects inside thecontainer, or a location on the pallet for stacking purposes) toidentify the task location 116. For example, the imaging devices 222 caninclude one or more cameras configured to generate images of the pickuparea and/or one or more cameras configured to generate images of thetask area (e.g., drop area). Based on the captured images, as describedbelow, the robotic system 100 can determine the start location 114, thetask location 116, the associated poses, and/or the confidence measures.

In some embodiments, the task can include scanning the target object112, such as for logging the item for shipping/receiving. To accomplishthe scanning portion of the task, the imaging devices 222 can includeone or more scanners (e.g., barcode scanners and/or QR code scanners)configured to scan the identification information during transfer (e.g.,between the start location 114 and the task location 116). Accordingly,the robotic system 100 can calculate a motion plan for presenting one ormore portions of the target object 112 to one or more of the scanners.

In some embodiments, for example, the sensors 216 can include positionsensors 224 (e.g., position encoders, potentiometers, etc.) configuredto detect positions of structural members (e.g., the robotic arms and/orthe end-effectors) and/or corresponding joints of the robotic system100. The robotic system 100 can use the position sensors 224 to tracklocations and/or orientations of the structural members and/or thejoints during execution of the task.

In some embodiments, for example, the sensors 216 can include contactsensors 226 (e.g., pressure sensors, force sensors, strain gauges,piezoresistive/piezoelectric sensors, capacitive sensors,elastoresistive sensors, and/or other tactile sensors) configured tomeasure a characteristic associated with a direct contact betweenmultiple physical structures or surfaces. The contact sensors 226 canmeasure the characteristic that corresponds to a grip of theend-effector (e.g., the gripper) on the target object 112. Accordingly,the contact sensors 226 can output a contact measure that represents aquantified measure (e.g., a measured force, torque, position, etc.)corresponding to a degree of contact or attachment between the gripperand the target object 112. For example, the contact measure can includeone or more force or torque readings associated with forces applied tothe target object 112 by the end-effector. Details regarding the contactmeasure are described below.

Initial Pose and Uncertainty Determinations

FIG. 3A, FIG. 3B, and FIG. 3C are illustrations of an object 302 invarious poses (e.g., a first pose 312, a second pose 314, and/or a thirdpose 316). A pose can represent a position and/or an orientation of theobject 302. In other words, the pose can include a translationalcomponent and/or a rotational component according to a grid systemutilized by the robotic system 100. In some embodiments, the pose can berepresented by a vector, a set of angles (e.g., Euler angles and/orroll-pitch-yaw angles), a homogeneous transformation, or a combinationthereof. The transformation of the object 302 can include arepresentation of a combination of the translational component, therotational component, a change therein, or a combination thereof. Therobotic system 100 can process an imaging output (e.g., a 2-dimensionalimage, a 3-dimensional image, a point cloud, and/or other imaging datafrom the imaging devices 222 of FIG. 2) to identify the pose of theobject 302. For example, the robotic system 100 can analyze the imagingoutput of one or more cameras directed to the pickup area to identifythe pose of the object 302 (e.g., the target object 112 of FIG. 1)located therein.

For identifying the pose, the robotic system 100 can first analyze theimaging data according to a pattern recognition mechanism and/or a setof rules to identify object outlines (e.g., perimeter edges orsurfaces). The robotic system 100 can further identify groupings ofobject outlines (e.g., according to predetermined rules and/or posetemplates) as corresponding to each unique instance of objects. Forexample, the robotic system 100 can identify the groupings of the objectoutlines that correspond to a pattern (e.g., same values or varying at aknown rate/pattern) in the color, the brightness, the depth/location, ora combination thereof across the object lines. Also, for example, therobotic system 100 can identify the groupings of the object outlinesaccording to predetermined shape/pose templates defined in the masterdata 252 of FIG. 2.

Once the object outlines are grouped, the robotic system 100 canidentify the pose of the object 302 relative to one or more coordinatesystems, such as according to a grid or a coordinate system used by therobotic system 100. For example, the robotic system 100 can identify oneor more surfaces, edges, and/or points of the object 302 and theorientation/location thereof according to the one or more coordinatesystems.

In some embodiments, the robotic system 100 can identify one or moreexposed surfaces (e.g., a first exposed surface 304, a second exposedsurface 306, etc.) of the object 302 in the imaging data. For example,the robotic system 100 can determine an outline shape and/or one or moredimensions (e.g., length, width, and/or height) of the object 302 fromthe imaging data according to the object outlines and the calibration ormapping data for the imaging devices 222. The robotic system 100 cancompare the determined dimensions to corresponding data in the masterdata 252 to identify the object 302. Further, the robotic system 100 canidentify an exposed surface as an object-top surface 322 or anobject-bottom surface 324 when dimensions of the exposed surface match alength and a width of the identified object. Also, the robotic system100 can identify the exposed surface as an object-peripheral surface 326when one of the dimensions of the exposed surface matches a height ofthe identified object.

In some embodiments, for example, the robotic system 100 can identifythe object 302 based on one or more markings (e.g., a letter, a number,a shape, a visual image, a logo, or a combination thereof) displayed onthe one or more exposed surfaces. The robotic system 100 can identifythe object 302 based on comparing the markings to one or morepredetermined images in the master data 252. For example, the roboticsystem 100 can include one or more images of a product name, a logo, adesign/image on the package surface, or a combination thereof in themaster data 252. The robotic system 100 can compare a portion of theimaging data (e.g., a portion within object outlines of the object 302)to the master data 252 to identify the object 302. The robotic system100 can similarly identify an orientation of the object 302 based onmatching the portion of the imaging data to a predetermined imagepattern that is unique for a surface.

As an illustrative example, FIG. 3A, FIG. 3B, and FIG. 3C illustrateexample imaging data corresponding to different poses of the object 302.FIG. 3A illustrates a first pose 312 where the first exposed surface 304(e.g., an exposed surface facing up) is the object-top surface 322 andthe second exposed surface 306 (e.g. an exposed surface generally facinga source of the imaging data) is one of the object-peripheral surfaces326.

In identifying the exposed surfaces, the robotic system 100 can processthe imaging data of FIG. 3A to measure the dimensions (e.g., number ofpixels) of the first exposed surface 304 and/or the second exposedsurface 306. The robotic system 100 can map the measurements in theimaging data to real-world dimensions using a predetermined cameracalibration or mapping function. The robotic system 100 can compare themapped dimensions to dimensions of known/expected objects in the masterdata 252 and identify the object based on matching the dimensions.Further, the robotic system 100 can identify that the first exposedsurface 304 is either the object-top surface 322 or the object-bottomsurface 324 since a pair of intersecting object edges that bound thefirst exposed surface 304 matches the length and the width of theidentified object. Similarly, the robotic system 100 can identify thesecond exposed surface 306 as the object-peripheral surface 326 sinceone of the object edges defining the second exposed surface 306 matchesthe height of the identified object.

In some embodiments, the robotic system 100 can process the imaging dataof FIG. 3A to identify one or more markings unique to a surface of theobject. For example, the robotic system 100 can include in the masterdata 252 one or more images and/or other visual characteristics (e.g.,color, dimension, size, etc.) of surfaces and/or unique markings of theobject as described above. As illustrated in FIG. 3A, the robotic system100 can identify the object as the object listed in the master data 252as having ‘A’ on the object-top surface 322. Accordingly, the roboticsystem 100 can further identify the first exposed surface 304 as theobject-top surface 322.

In some embodiments, the robotic system 100 can include in the masterdata 252 information regarding an object identifier 332 (e.g., acomputer-readable visual identifier, such as a bar code or a QR code,that is unique to the object 302). For example, the master data 252 caninclude the image and/or coded message of the object identifier 332, anidentifier location 334 relative to a surface and/or a set of edges, oneor more visual characteristics thereof, or a combination thereof. Asillustrated in FIG. 3A, the robotic system 100 can identify the secondexposed surface 306 as the object-peripheral surface 326 based on thepresence of the object identifier 332 on the surface and/or the locationthereof matching the identifier location 334.

FIG. 3B illustrates a second pose 314 where the object 302 is rotated 90degrees about a vertical axis along a direction B in FIG. 3A. Forexample, a reference point ‘a’ of the object 302 can be in the lowerleft corner in FIG. 3A and in the lower right corner in FIG. 3B.Accordingly, in comparison to the first pose 312, the object-top surface322 can be seen in the imaging data in a different orientation and/orthe object-peripheral surface 326 having the object identifier 332 canbe hidden from view.

The robotic system 100 can identify the different poses based on aspecial orientation of one or more identifying visual features. Forexample, the robotic system 100 can determine the first pose 312 and/orthe third pose 316 when a dimension matching a known length of an objectextends horizontally in the imaging data, a dimension matching a knownheight of the object extends vertically in the imaging data, and/ordimension that matches a known width of the object extends along a depthaxis in the imaging data. Similarly, the robotic system 100 candetermine the second pose 314 when the dimension matching the widthextends horizontally, the dimension matching the height extendsvertically, and/or the dimension matching the length extends along thedepth axis. Also, for example, the robotic system 100 can determine thatthe object 302 is in the first pose 312 or the second pose 314 based onan orientation of a visible marking, such as ‘A’ shown in FIG. 3A andFIG. 3B. Also, for example, the robotic system 100 can determine thatthe object 302 is in the first pose 312 according to visible markingsseen in a combination of surfaces, such as when the object identifier332 is visible with (i.e., on different surfaces) the marking ‘A.’

FIG. 3C illustrates a third pose 316 where the object 302 is rotated 180degrees about a horizontal axis along a direction C in FIG. 3A. Forexample, a reference point ‘a’ of the object 302 can be in the lowerleft front corner in FIG. 3A and in the upper left back corner in FIG.3C. Accordingly, in comparison to the first pose 312, the first exposedsurface 304 can be the object-bottom surface 324, and both theobject-top surface 322 and the object-peripheral surface 326 having theobject identifier 332 can be hidden from view.

As described above, the robotic system 100 can identify that the object302 is in either the first pose 312 or the third pose 316 based on thedimensions. The robotic system 100 can identify that the object 302 isin the first pose 312 when a top surface marker (e.g., ‘A’) is visible.Also, the robotic system 100 can identify that the object 302 is in thethird pose 316 when a bottom-surface marker (e.g., an instance of theobject identifier 332) is visible.

In determining the pose of the object 302, real-world conditions mayaffect the accuracy of the determination. For example, lightingconditions may reduce visibility of surface markings, such as due toreflections and/or shadows. Also, an actual orientation of the object302 may reduce an exposure or viewing angle of one or more presentedsurfaces such that any markings thereon may be unidentifiable. As such,in some embodiments, the robotic system 100 can calculate a confidencemeasure associated with a determined pose. The confidence measure canrepresent a measure of accuracy of the determined pose. In someembodiments, the confidence measure can correspond to a likelihood thatthe determined pose matches the actual pose of the object 302.

In some embodiments, for example, the robotic system 100 can calculatethe confidence measure based on a measure of the match used indetermining the pose. For example, the robotic system 100 can calculatethe confidence measure based on a certainty interval associated with themeasurements of dimensions in the image. In some embodiments, thecertainty interval can increase as a distance between the object 302 andthe imaging source (e.g., the imaging devices 222 of FIG. 2) decreasesand/or when a measured edge of the object 302 is closer to beingorthogonal to a direction radiating from the imaging source and fartheraway from being parallel to the radiating direction. Also, for example,the robotic system 100 can calculate the confidence measure based on adegree of match between a marker or a design in the imaging data to aknown marker/design in the master data 252. In some embodiments, therobotic system 100 can use an overlap or a deviation measure between theimaging data or a portion thereof and the predetermined markers/images.The robotic system 100 can identify the object and/or the orientationaccording to the greatest overlap and/or the lowest deviation measure,such as for a minimum mean square error (MMSE) mechanism. Moreover, therobotic system can calculate the confidence measure based on theresulting overlap/deviation measure. As described in more detail below,the robotic system 100 can calculate a motion path according to theconfidence measure. In other words, the robotic system 100 can move theobject 302 differently according to the confidence measure.

System Operation

FIG. 4 is a top view illustrating an example task 402 executed by therobotic system 100 in accordance with one or more embodiments of thepresent disclosure. As described above, the task 402 can represent asequence of actions executed by the robotic system 100 (e.g., by one ofthe units described above, such as the transfer unit 104 of FIG. 1) toachieve a goal. As illustrated in FIG. 4, for example, the task 402 caninclude moving the target object 112 from the start location 114 (e.g.,a location on/in a receiving pallet or bin) to the task location 116(e.g., a location on/in a sorted pallet or bin). The task 402 canfurther include scanning the target object 112 while moving from thestart location 114 to the task location 116. Accordingly, the roboticsystem 100 can update the object tracking data 254 of FIG. 2 accordingto the scanned information, such as by adding, removing, and/orverifying the scanned object from the object tracking data 254.

In some embodiments, the robotic system 100 can image a predeterminedarea to identify and/or locate the start location 114. For example, therobotic system 100 can include a source scanner 412 (i.e., an instanceof the imaging devices 222 of FIG. 2) directed at a pickup area, such asan area designated for a sourcing pallet or bin and/or a region on areceiving side of the conveyor belt. The robotic system 100 can use thesource scanner 412 to generate imaging data (e.g., a captured imageand/or a point cloud) of the designated area. The robotic system 100(via, e.g., the processors 202 of FIG. 2) can implement computer visionprocesses for the imaging data to identify the different objects (e.g.,boxes or cases) located in the designated area. Details of the objectidentification are described below.

From the recognized objects, the robotic system 100 can select (e.g.,according to a predetermined sequence or set of rules and/or templatesof object outlines) one object as the target object 112 for an executionof the task 402. For the selected target object 112, the robotic system100 can further process the imaging data to determine the start location114 and/or an initial pose. Details of the selection and thelocation/pose determination are described below.

The robotic system 100 can further image and process anotherpredetermined area to identify the task location 116. In someembodiments, for example, the robotic system 100 can include anotherinstance of the imaging devices 222 (not shown) configured to generateimaging data of a placement area, such as an area designated for asorted pallet or bin and/or a region on a sending side of the conveyorbelt. The imaging result can be processed (via, e.g., the processors202) to identify the task location 116 and/or a corresponding pose forplacing the target object 112. In some embodiments, the robotic system100 can identify (based on or not based on the imaging result) the tasklocation 116 according to a predetermined sequence or set of rules forstacking and/or arranging multiple objects.

In some embodiments, the task 402 can include scanning (e.g., scanningthe object identifier 332 of FIG. 3A and/or FIG. 3C) the target object112 for product logging purposes and/or for further identifying thetarget object 112. For example, the robotic system 100 can include oneor more object scanners 416 (e.g., further instances of the imagingdevices 222, such as barcode scanners or QR code scanners) configured toscan the target object 112, typically at one or more locations betweenthe pickup area and the placement area. In some embodiments, the objectscanners 416 can face horizontal directions to scan marks that areadjacent to the scanners (e.g., at a height corresponding to that of thecorresponding scanner(s)) and on vertically oriented surfaces. In someembodiments, the object scanners 416 can face vertical directions toscan marks that are above/below the scanner and on horizontally orientedsurfaces. In some embodiments, the object scanners 416 can face eachother such that they can scan opposite sides of the object that isplaced between the object scanners 416. According to the location and/orscanning direction of the object scanners 416, the robotic system 100can manipulate the target object 112 to place the target object 112 at apresentation location and/or according to a presentation pose forscanning one or more surfaces/portions of the target object 112 with theobject scanners 416.

Using the identified start location 114 and/or the task location 116,the robotic system 100 can operate one or more structures (e.g., arobotic arm 414 and/or the end-effector) of a corresponding unit (e.g.,the transfer unit 104) to execute the task 402. Accordingly, the roboticsystem 100 (via, e.g., the processors 202) can calculate (via, e.g.,motion planning rules or algorithms) a motion plan that corresponds toone or more actions that will be implemented by the corresponding unitto execute the task 402. For example, the motion plan for the transferunit 104 can include positioning the end-effector at an approachlocation (e.g., a location/position for placing an end-effector tocontact and grip the target object 112), gripping the target object 112,lifting the target object 112, transferring the target object 112 fromabove the start location 114 to the presentation location/pose for thescanning operation, transferring the target object 112 from thepresentation location to above the task location 116, lowering thetarget object 112, and releasing the target object 112.

In some embodiments, the robotic system 100 can calculate the motionplan by determining a sequence of commands and/or settings for one ormore of the actuation devices 212 of FIG. 2 that operate the robotic arm414 and/or the end-effector. For example, the robotic system 100 can usethe processors 202 to calculate the commands and/or settings of theactuation devices 212 for manipulating the end-effector and the roboticarm 414 to place the end-effector (e.g., a gripper) at the approachlocation about the start location 114, engage and grab the target object112 with the end-effector, place the end-effector at a scanning position(e.g., a designated location and/or orientation) about the presentationlocation, place the end-effector at a particular location about the tasklocation 116, and release the target object 112 from the end-effector.The robotic system 100 can execute the actions for completing the task402 by operating the actuation devices 212 according to the determinedsequence of commands and/or settings.

In some embodiments, the robotic system 100 can derive the motion planbased on a confidence measure that represents a measure of certainty orlikelihood that the determined pose matches the actual real-world poseof the target object 112. For example, the robotic system 100 can placethe end-effector at different locations for pickup, such as for grippingor covering different surfaces, calculate different presentationlocations/poses for the target object 112, or a combination thereofaccording to the confidence measure.

As an illustrative example, the target object 112 can be the object 302of FIG. 3A placed in the first pose 312 of FIG. 3A (i.e., the object-topsurface 322 of FIG. 3A generally facing up and exposed). When theconfidence measure is high (i.e., a degree of certainty above athreshold, representing that the determined pose is more likelyaccurate), the robotic system 100 can calculate a first motion plan 422that includes a first approach location 432 and a first presentationlocation 442. For example, since there is sufficient certainty that theobject-top surface 322 is facing upward (i.e., the object-bottom surface324 of FIG. 3C with the object identifier 332 of FIG. 3C is facingdownward), the robotic system 100 may calculate the first motion plan422, which includes the first approach location 432, for placing theend-effector directly over the object-top surface 322. Accordingly, therobotic system 100 can grip the target object 112 with the end-effectorcontacting/covering the object-top surface 322 such that theobject-bottom surface 324 is exposed. Also, the robotic system 100 cancalculate the first motion plan 422 that includes the first presentationlocation 442 for the target object 112 directly over an upward-facingscanner for scanning the object identifier 332 located on theobject-bottom surface 324.

In contrast, when the confidence measure is low (i.e., a degree ofcertainty below a threshold, representing that the determined pose isless likely accurate), the robotic system 100 can calculate a secondmotion plan 424 (i.e., different from the first motion plan 422) thatincludes a second approach location 434 and one or more secondpresentation locations 444. For example, the robotic system 100 canmeasure and compare the dimensions of the target object 112 anddetermine (e.g., when the certainty levels of the measurements exceed apredetermined threshold) that the object is in either the first pose 312of FIG. 3A or the third pose 316 of FIG. 3C. However, the robotic system100 may have difficulties imaging/processing marks printed on thesurface of the target object 112 such that the confidence measureassociated with the determined pose is below a threshold. In otherwords, the robotic system 100 may not be sufficiently certain whetherthe upward facing exposed surface is the object-top surface 322 (e.g.,the first pose 312) or the object-bottom surface 324 (e.g., the thirdpose 316).

Due to the higher degree of uncertainty, the robotic system 100 maycalculate the second motion plan 424 that includes the second approachlocation 434 for placing the end-effector adjacent (e.g., aligned withand/or facing a direction parallel to the object-top surface 322 and/orthe object-bottom surface 324) to one of the object-peripheral surfaces326 of FIG. 3A. Accordingly, the robotic system 100 can grip the targetobject 112 with the end-effector contacting/covering one of theobject-peripheral surfaces 326 and exposing both the object-top surface322 and the object-bottom surface 324. The robotic system 100 cansimultaneously or sequentially present or place the object-top surface322 and the object-bottom surface 324 before (e.g., in the scanningfields of and/or facing) the object scanners 416. When the target object112 is in place for the scan, the robotic system 100 can operate theobject scanners 416 (e.g., at least the scanners facing the object-topsurface 322 and the object-bottom surface 324) to simultaneously and/orsequentially scan the presented surfaces and capture the objectidentifier(s) 332 thereon.

Also, the robotic system 100 can calculate the second motion plan 424,which includes the second presentation location(s) 444, for placing aninitially downward-facing surface (the object-bottom surface 324)horizontally and directly over an upward-facing scanner and/or forplacing an initially upward-facing surface (the object-top surface 322)vertically and directly in front of a horizontally facing scanner. Thesecond motion plan 424 can include a reorienting/rotating action (e.g.,as represented by a dotted-unfilled circle) for providing the twopresentation locations/poses, thereby scanning both opposing top/bottomsurfaces using orthogonally directed scanners. For example, the roboticsystem 100 can sequentially present the object-top surface 322 to anupward-facing scanner and scan, and then rotate the target object 112 90degrees to present the object-bottom surface 324 to ahorizontally-facing scanner for scanning. In some embodiments, thereorienting/rotating action can be conditional such that the roboticsystem 100 implements the corresponding commands when the first scan isunsuccessful in reading the object identifier 332.

Alternatively, as an example, the robotic system 100 can calculate amotion plan (not shown) for gripping/covering one of theobject-peripheral surfaces 326 along the width of the target object 112when the confidence measure is low. The robotic system 100 can move thetarget object 112 between a horizontally opposing pair of the objectscanners 416 to present the object-peripheral surfaces 326 along thelength of the target object 112 to scan the object identifier 332 on oneof such peripheral surfaces (e.g., as shown in FIG. 3A). Detailsregarding the motion planning based on the confidence measure aredescribed below.

In executing the actions for the task 402, the robotic system 100 cantrack a current location (e.g., a set of coordinates corresponding to agrid used by the robotic system 100) and/or a current pose of the targetobject 112. For example, the robotic system 100 (via, e.g., theprocessors 202) can track the current location/pose according to datafrom the position sensors 224 of FIG. 2. The robotic system 100 canlocate one or more portions of the robotic arm 414 (e.g., the structuralmembers and/or the joints thereof) in the kinetic chain according to thedata from the position sensors 224. The robotic system 100 can furthercalculate the location/pose of the end-effector, and thereby the currentlocation of the target object 112 held by the end-effector, based on thelocation and orientation of the robotic arm 414. Also, the roboticsystem 100 can track the current location based on processing othersensor readings (e.g., force readings or accelerometer readings), theexecuted actuation commands/settings and/or associated timings, or acombination thereof according to a dead-reckoning mechanism.

Operational Flow

FIG. 5A is a flow diagram for a method 500 of operating the roboticsystem 100 of FIG. 1 in accordance with one or more embodiments of thepresent disclosure. The method 500 can be for executing the task 402 ofFIG. 4 according to a confidence measure associated with an initial posedetermination. The method 500 can be for deriving/calculating andimplementing a motion plan based on the confidence measure. The method500 can be implemented based on executing the instructions stored on oneor more of the storage devices 204 of FIG. 2 with one or more of theprocessors 202 of FIG. 2.

At block 501, the robotic system 100 can identify scanning fields of oneor more of the imaging devices 222 of FIG. 2. For example, the roboticsystem 100 (via, e.g., one or more of the processors 202) can identifyspaces that can be scanned by one or more of the imaging devices 222,such as the source scanner 412 of FIG. 4 and/or the object scanners 416of FIG. 4. In some embodiments, the robotic system 100 can identify thescanning fields that are oriented in opposite directions and/ororthogonal directions according to orientations of the object scanners416. As illustrated in FIG. 4, in some embodiments, the object scanners416 can be arranged opposite each other and/or facing each other, suchas across a horizontal direction or across a vertical direction. Also,in some embodiments, the object scanners 416 can be arrangedperpendicular to each other, such as one facing up or down and anotherfacing a horizontal direction.

In some embodiments, for example, the robotic system 100 can identifythe scanning fields according to the master data 252. The master data252 can include grid locations, coordinates, and/or other markersrepresenting the imaging devices 222 and/or the corresponding scanningfields. The master data 252 can be predetermined according to a layoutand/or a physical placement of the imaging devices 222, the capabilitiesof the imaging devices 222, environmental factors (e.g., lightingconditions and/or obstacles/structures), or a combination thereof. Insome embodiments, the robotic system 100 can implement a calibrationprocess to identify the scanning fields. For example, the robotic system100 can use the transfer unit 104 of FIG. 1 to place a known mark orcode at a set of locations and determine whether the correspondingimaging device accurately scans the known mark. The robotic system 100can identify the scanning fields based on the locations of the knownmark that resulted in accurate scanning results.

At block 502, the robotic system 100 can scan designated areas. In someembodiments, the robotic system 100 can use (via, e.g., commands/promptssent by the processors 202) one or more of the imaging devices 222(e.g., the source scanner 412 of FIG. 4 and/or other area scanners) togenerate imaging data (e.g., captured digital images and/or pointclouds) of one or more designated areas, such as the pickup area and/orthe drop area (e.g., the source pallet/bin/conveyor and/or the taskpallet/bin/conveyor). The imaging data can be communicated from theimaging devices 222 to the one or more processors 202. Accordingly, oneor more of the processors 202 can receive the imaging data thatrepresents the pickup area (e.g., including objects before execution ofthe task) and/or the drop area (e.g., including objects after executionof the task) for further processing.

At block 504, the robotic system 100 can identify the target object 112of FIG. 1 and associated locations (e.g., the start location 114 of FIG.1 and/or the task location 116 of FIG. 1) and/or orientations (e.g.,initial pose). In some embodiments, for example, the robotic system 100(via, e.g., the processors 202) can analyze the imaging data accordingto a pattern recognition mechanism and/or a set of rules to identifyobject outlines (e.g., perimeter edges and/or surfaces). The roboticsystem 100 can further identify groupings of object outlines (e.g.,according to predetermined rules and/or pose templates) and/or surfacesas corresponding to each unique instance of objects. For example, therobotic system 100 can identify the groupings of the object outlinesthat correspond to a pattern (e.g., same values or varying at a knownrate/pattern) in the color, the brightness, the depth/location, or acombination thereof across the object lines. Also, for example, therobotic system 100 can identify the groupings of the object outlinesand/or surfaces according to predetermined shape/pose templates, images,or a combination thereof defined in the master data 252.

From the recognized objects in the pickup location, the robotic system100 can select (e.g., according to a predetermined sequence or set ofrules and/or templates of object outlines) one as the target object 112.For example, the robotic system 100 can select the target object 112 asthe object located on top, such as according to the point cloud(representing the distances/positions relative to a known location ofthe source scanner 412). Also, for example, the robotic system 100 canselect the target object 112 as the object that is located at acorner/edge and has two or more surfaces that are exposed/shown in theimaging results. Further, the robotic system 100 can select the targetobject 112 according to a predetermined pattern or sequence (e.g., leftto right, nearest to furthest, etc., relative to a reference location).

For the selected target object 112, the robotic system 100 can furtherprocess the imaging result to determine the start location 114 and/orthe initial pose. For example, the robotic system 100 can determine thestart location 114 by mapping a location (e.g., a predeterminedreference point for the determined pose) of the target object 112 in theimaging result to a location in the grid used by the robotic system 100.The robotic system 100 can map the locations according to apredetermined calibration map.

In some embodiments, the robotic system 100 can process the imagingresults of the drop areas to determine open spaces between objects. Therobotic system 100 can determine the open spaces based on mapping theobject lines according to a predetermined calibration map that mapsimage locations to real-world locations and/or coordinates used by thesystem. The robotic system 100 can determine the open spaces as thespace between the object lines (and thereby object surfaces) belongingto different groupings/objects. In some embodiments, the robotic system100 can determine the open spaces suitable for the target object 112based on measuring one or more dimensions of the open spaces andcomparing the measured dimensions to one or more dimensions of thetarget object 112 (e.g., as stored in the master data 252 of FIG. 2).The robotic system 100 can select one of the suitable/open spaces as thetask location 116 according to a predetermined pattern (e.g., left toright, nearest to furthest, bottom to top, etc., relative to a referencelocation).

In some embodiments, the robotic system 100 can determine the tasklocation 116 without or in addition to processing the imaging results.For example, the robotic system 100 can place the objects at theplacement area according to a predetermined sequence of actions andlocations without imaging the area. Also, for example, the roboticsystem 100 can process the imaging results for performing multiple tasks(e.g., transferring multiple objects, such as for objects located on acommon layer/tier of a stack).

At block 522, for example, the robotic system 100 can determine aninitial pose (e.g., an estimate of a resting orientation of the targetobject 112 at the pickup location) based on processing the imaging data(e.g., the imaging data from the source scanner 412). In someembodiments, the robotic system 100 can determine the initial pose ofthe target object 112 based on comparing (e.g., comparing pixel values)the object outlines to outlines in predetermined pose templates of themaster data 252. The predetermined pose templates can include, e.g.,different potential arrangements of the object outlines according tocorresponding orientations of expected objects. The robotic system 100can identify the set of object outlines (e.g., edges of an exposedsurface, such as the first exposed surface 304 of FIG. 3A and/or FIG. 3Cand/or the second exposed surface 306 of FIG. 3A) that were previouslyassociated with the object selected as the target object 112. Therobotic system 100 can determine the initial pose based on selecting oneof the pose templates that corresponds to a lowest difference measurebetween the compared object outlines.

In some embodiments, for further example, the robotic system 100 candetermine the initial pose of the target object 112 based on physicaldimensions of the target object 112. The robotic system 100 can estimatephysical dimensions of the target object 112 based on the dimensions ofthe exposed surfaces captured in the imaging data. The robotic system100 can measure a length and/or an angle for each object outline in theimaging data and then map or convert the measured length to real-worldor standard lengths using a calibration map, a conversion table orprocess, a predetermined equation, or a combination thereof. The roboticsystem 100 can use the measured dimensions to identify the target object112 and/or the exposed surface(s) corresponding to the physicaldimensions. The robotic system 100 can identify the object and/or theexposed surface(s) based on comparing the estimated physical dimensionsto a set of known dimensions (e.g., height, length, and/or width) ofobjects and their surfaces in the master data 252. The robotic system100 can use the matched set of dimensions to identify the exposedsurface(s) and the corresponding orientation. For example, the roboticsystem 100 can identify the exposed surface as either the object-topsurface 322 of FIG. 3A or the object-bottom surface 324 of FIG. 3B(e.g., a pair of opposing surfaces) when the dimensions of the exposedsurface match a length and a width for an expected object. Based on theorientation of the exposed surface, the robotic system 100 can determinethe initial pose (e.g., either the first pose 312 or the third pose 316when the exposed surface is facing upward).

In some embodiments, for example, the robotic system 100 can determinethe initial pose of the target object 112 based on a visual image of oneor more surfaces of the target object 112 and/or one or more markingsthereof. The robotic system 100 can compare the pixel values within aset of connected outlines to predetermined marking-based pose templatesof the master data 252. The marking-based pose templates can include,e.g., one or more unique markings of expected objects in variousdifferent orientations. The robotic system 100 can determine the initialpose based on selecting one of the surfaces, the surface orientations,and/or the corresponding poses that results in a lowest differencemeasure for the compared images.

At block 524, the robotic system 100 can calculate a confidence measureassociated with the initial pose. The confidence measure can represent ameasure of certainty or likelihood that the initial pose matches theactual real-world pose of the target object 112. In some embodiments,the robotic system 100 can calculate the confidence measure as a part ofdetermining the initial pose. For example, the confidence measure cancorrespond to the difference measure between the object outlines and theoutlines in the selected template described above. Also, for example,the confidence measure can correspond to a tolerance level associatedwith the estimated physical dimensions and/or the angles describedabove. Also, for example, the confidence measure can correspond to thedifference measure between a visual marking in the imaging data and thetemplate images described above.

At block 506, the robotic system 100 can calculate a motion plan (e.g.,the first motion plan 422 of FIG. 4 and/or the second motion plan 424 ofFIG. 4) for executing the task 402 for the target object 112. Forexample, the robotic system 100 can calculate the motion plan based oncalculating a sequence of commands or settings, or a combinationthereof, for the actuation devices 212 of FIG. 2 that will operate therobotic arm 414 of FIG. 4 and/or the end-effector. For some tasks, therobotic system 100 can calculate the sequence and the setting valuesthat will manipulate the robotic arm 414 and/or the end-effector totransfer the target object 112 from the start location 114 to the tasklocation 116. The robotic system 100 can implement a motion planningmechanism (e.g., a process, a function, an equation, an algorithm, acomputer-generated/readable model, or a combination thereof) configuredto calculate a path in space according to one or more constraints,goals, and/or rules. For example, the robotic system 100 can use A*algorithm, D* algorithm, and/or other grid-based searches to calculatethe path through space for moving the target object 112 from the startlocation 114 to the task location 116 through one or more presentationposes/locations (e.g., one or more corresponding scanning locations forthe end-effector). The motion planning mechanism can use a furtherprocess, function, or equation, and/or a mapping table, to convert thepath into the sequence of commands or settings, or combination thereof,for the actuation devices 212. In using the motion planning mechanism,the robotic system 100 can calculate the sequence that will operate therobotic arm 414 and/or the end-effector and cause the target object 112to follow the calculated path.

In some embodiments, the robotic system 100 can selectivelycalculate/derive the motion plan based on the confidence measure. Therobotic system 100 can calculate the motion plan that includes anapproach location (e.g., the first approach location 432 of FIG. 4and/or the second approach location 434 of FIG. 4), one or more scanninglocations (e.g., the first presentation location 442 of FIG. 4 and/orthe second presentation location 444 of FIG. 4), or a combinationthereof according to the confidence measure. For example, the roboticsystem 100 can calculate the approach location and/or the scanninglocations according to a metric (e.g., a performance metric and/or ascanning metric) based on an outcome of comparing the confidence measureto a sufficiency threshold. The scanning location can be for placing theend-effector so as to present one or more surfaces of the target object112 before (i.e., in the scanning field of) one or more correspondingobject scanners 416 that are to scan the one or more object identifiers332.

At block 532, the robotic system 100 (via, e.g., the processors 202) cancalculate a set of available approach locations. The available approachlocations can correspond to open or non-occupied spaces about the startlocation 114 sufficient for placing the end-effector. As describedfurther below, the robotic system 100 can place the end-effector at aselected approach location for contacting and gripping the target object112 without disturbing other objects.

In some embodiments, for example, the robotic system 100 can calculatethe set of available approach locations based on calculating separationdistances between object outlines of the target object 112 and those ofadjacent objects. The robotic system 100 can compare the separationdistances to a predetermined set of distances that correspond to aphysical size/shape of the end-effector and/or different orientationsthereof. The robotic system can identify each of the available approachlocations when the corresponding separation distances exceed thepredetermined set of distances corresponding to the size of theend-effector.

At decision block 534, the robotic system 100 can compare the confidencemeasure to one or more sufficiency conditions (e.g., one or morethresholds) to determine whether or not they are satisfied. When theconfidence measure satisfies the sufficiency condition (e.g., theconfidence measure exceeds the required threshold), such as illustratedat block 536, the robotic system 100 can calculate the motion plan(e.g., the first motion plan 422) based on a performance metric. In someembodiments, when the confidence measure satisfies the sufficiencycondition, the robotic system 100 can assume that the initial pose iscorrect and calculate the motion plan without considering a scanningmetric that corresponds to a likelihood for scanning at least one objectidentifier and/or a possibility that the initial pose may be inaccurate.

As an illustrative example, in some embodiments, the robotic system 100can calculate candidate plans at block 542. The candidate plans can eachbe an instance of a motion plan that corresponds to a unique combinationof an available approach location and a scanning location (e.g.,corresponding presentation location/orientation for the target object112). In some embodiments, the robotic system 100 can calculate theidentifier location(s) 334 according to the initial pose, such as byrotating the identifier location(s) 334 or a corresponding model/pose inthe master data 252. The robotic system 100 can eliminate availableapproach locations that would have the end-effector cover (e.g., bedirectly over, in front of, and/or within a threshold distance from) theidentifier location(s) 334.

The robotic system 100 can calculate a candidate plan for each remainingavailable approach location in the set (e.g., calculation result ofblock 532). For each candidate plan, the robotic system 100 can furthercalculate a unique scanning location according to the available approachlocation. In some embodiments, the robotic system 100 can calculate thescanning location based on rotating and/or translating a model of thetarget object 112, such that the surface corresponding to the identifierlocation 334 is in the scanning field and faces the corresponding objectscanner. The robotic system 100 can rotate and/or translate the modelaccording to a predetermined process, equation, function, etc.

At block 544, the robotic system 100 can calculate a performance metricfor each candidate plan. The robotic system 100 can calculate theperformance metric that corresponds to a throughput rate for completingthe task 402. For example, the performance metric can be associated witha distance traversed by the target object 112, an estimated transferduration, a number of commands and/or setting changes for the actuationdevices 212, a completion rate (i.e., complementary to a piece-lossrate), or a combination thereof for the candidate plan. The roboticsystem 100 can calculate the corresponding values for the candidatemotion plan using one or more measured or known data (e.g.,acceleration/speed associated with settings/commands and/or piece-lossrate associated with a grip surface and/or a maneuver) and apredetermined calculation process, equation, function, etc.

At block 546, the robotic system 100 can select the candidate plan(i.e., along with the corresponding approach location) with the maximumperformance metric as the motion plan. For example, the robotic system100 can select as the motion plan the candidate plan that corresponds tothe highest completion rate, the shortest travel distance, the lowestnumber of commands and/or setting changes, the fastest transferduration, or a combination thereof among the set of candidate plans.Accordingly, the robotic system 100 can select the available approachlocation in the set that corresponds to the highest performance metricas the approach location.

In comparison, the robotic system 100 can calculate the candidate planaccording to a different measure when the confidence measure fails tosatisfy the sufficiency condition (e.g., the confidence measure is belowthe required threshold). In some embodiments, as illustrated at block538, the robotic system 100 can calculate the motion plan (e.g., thesecond motion plan 424) based on a scanning metric. The scanning metricis a value (e.g., a binary value or a non-binary score/percentage) thatcorresponds to a likelihood that at least one of the object identifiers332 remains uncovered by the end-effector and is scannable, regardlessof whether the initial pose is accurate. In some embodiments, forexample, the robotic system 100 can prioritize the scanning metric(e.g., satisfy first and/or give it a heavier weight) over theperformance metrics when the confidence measure fails to satisfy thesufficiency condition. Accordingly, the robotic system 100 can calculatethe motion plan that includes one or more scanning locations forpresenting (i.e., in the scanning field and/or facing the correspondingscanner) the at least one uncovered object identifier before one or moreof the object scanners. In illustrating the process in further detail,FIG. 5B is a flow diagram 538 for selectively calculating motion plans(e.g., one or more locations for the end-effector) based on scanningmetrics in accordance with one or more embodiments of the presentdisclosure.

In some embodiments, calculating the motion plan based on a scanningmetric can include calculating a set of exposed identifier locations asillustrated in block 552. The robotic system 100 can calculate the setof exposed identifier locations (e.g., the identifier locations 334 thatcan remain scannable with the end-effector in gripping position)relative to the initial pose. The robotic system 100 can calculate theexposed identifier locations for each available approach location. Theexposed identifier location can correspond to locations of the objectidentifiers 332 that remain uncovered with the end-effector at thecorresponding approach location according to a hypothesis that theinitial pose is accurate.

As described above (for block 542), in some embodiments, the master data252 can include a computer model or a template (e.g., offset measuresrelative to one or more object edges and/or images) that describes theidentifier locations 334 for each of the expected objects. The roboticsystem 100 can calculate the set of exposed identifier locations basedon rotating and/or translating the predetermined model/template in themaster data 252 to match the initial pose. In some embodiments, therobotic system 100 can eliminate available approach locations that wouldhave the end-effector cover (e.g., be directly over, in front of, and/orwithin a threshold distance from) the identifier locations 334. In otherwords, the robotic system 100 can eliminate the approach locations thatare directly over, in front of, and/or within a threshold distance fromthe identifier locations 334.

At block 554, the robotic system 100 can calculate a set of alternativeidentifier locations. The robotic system 100 can calculate the set ofthe alternative identifier locations (e.g., the identifier locations334) for poses alternative to the initial pose. For each availableapproach location, the robotic system 100 can calculate alternativeposes, and for each alternative pose, the robotic system 100 cancalculate the alternative identifier locations. As such, the alternativeidentifier locations can correspond to locations of the objectidentifiers 332 that remain uncovered with the end-effector at thecorresponding approach location according to a hypothesis that theinitial pose is inaccurate. As described above for the exposedidentifier locations, the robotic system 100 can calculate thealternative identifier locations based on rotating and/or translatingthe predetermined model/template in the master data 252 according to thealternative pose.

At block 556, the robotic system 100 can calculate an exposurelikelihood for each of the approach locations, each of the alternativeposes, each of the object identifiers, or a combination thereof. Theexposure likelihood represents a likelihood that one or more objectidentifiers would remain exposed and scannable with the end-effectorgripping the target object 112 from the corresponding approach location.The exposure likelihood can represent both the scenario that the initialpose is accurate and the scenario that the initial pose is inaccurate.In other words, the exposure likelihood can represent the likelihoodthat one or more object identifiers would remain exposed and scannableeven if the initial pose is inaccurate.

In some embodiments, for example, the robotic system 100 can calculatethe exposure likelihood as a conditional probability, such as aprobabilistic value corresponding to a particular condition (e.g., aunique instance of the approach location, the alternative pose, theobject identifier, or a combination thereof). In some embodiments, therobotic system 100 can calculate the exposure likelihood based oncombining (via, e.g., adding and/or multiplying) the conditionalprobability with a probability/likelihood that the particular conditionis true (e.g., a value similar to the confidence measure). In someembodiments, the robotic system 100 can calculate the exposurelikelihood based on adding the probabilities for each likely exposedidentifier when multiple identifiers would be exposed for the consideredapproach location and/or the considered pose.

In some embodiments, the robotic system 100 can calculate the exposurelikelihood based on combining the probabilistic values based on theexposed identifier locations and the alternative identifier locations,such as for each potential pose for a considered approach location. Forexample, the robotic system 100 can calculate the exposure likelihoodusing the probabilities for the exposed identifier locations and thealternative identifier locations with opposing signs (e.g., positive andnegative). The robotic system 100 can calculate the exposure likelihoodbased on adding the magnitudes of the two probabilities and/or addingthe probabilities with the signs. The overall magnitude can represent anoverall likelihood that one or more object identifiers would remainscannable, and the signed/vectored likelihood can represent a likelihoodthat one or more object identifiers would remain scannable even if theinitial pose was inaccurate. Accordingly, an approach position would beideal when the overall magnitude is higher, and the signed/vectoredlikelihood is closer to zero, such as for representing similar chancesthat an object identifier would be scannable regardless of the accuracyfor the initial pose.

At block 558, the robotic system 100 can select an approach location. Insome embodiments, the robotic system 100 can select as the approachlocation the available approach location that includes uncoveredidentifier locations in both an exposed identifier set (e.g., a set ofestimated locations of the object identifiers according to a hypothesisthat the initial pose is correct) and an alternative identifier set(e.g., one or more sets of estimated locations of the object identifiersaccording to a hypothesis that the initial pose is incorrect). In otherwords, the robotic system 100 can select the approach location thatwould leave at least one object identifier exposed and scannableregardless of the accuracy of the initial pose. In some embodiments, therobotic system 100 can select as the approach location the availableapproach location that corresponds to the exposure likelihood matchingand/or closest to a targeted condition, such as the highest overallmagnitude and/or the signed/vectored likelihood that is closer to zero.

In some embodiments, the robotic system 100 can calculate a scanlikelihood (e.g., a likelihood that an exposed object identifier wouldbe successfully scanned) based on the exposure likelihood. For example,the robotic system 100 can combine the exposure likelihood with anevaluation value (e.g., a tracked rate of successful scans, a physicalsize, and/or an identifier type) associated with the correspondingexposed object identifier. The robotic system 100 can select as theapproach location the available approach location that corresponds tothe highest scan likelihood.

In some embodiments, the robotic system 100 can compare the exposedidentifier set to the alternative identifier set to determine whetherthe exposed identifier set and the alternative identifier set includelocations on opposing surfaces of the target object 112 (e.g., betweenthe first pose 312 and the third pose 316). Accordingly, the roboticsystem 100 can select an available approach location that corresponds toa third surface (e.g., one of the object-peripheral surfaces 326) thatis orthogonal to the two opposing surfaces.

At block 560, such as when the confidence measure fails to satisfy thesufficiency threshold, the robotic system 100 can calculate candidatemotion plans based on the selected approach location. The robotic system100 can calculate the candidate motion plans that include one or morescanning locations for the end-effector that correspond to one or morepresentation locations/orientations that place the object identifiers inboth the exposed identifier set and the alternative identifier set. Inother words, the robotic system 100 can calculate the candidate motionplans that can scan the target object 112 regardless of the accuracy ofthe initial pose.

In some embodiments, the robotic system 100 can calculate the candidatemotion plans that account for the identifier locations in both theexposed identifier set and the alternative identifier set. For example,the robotic system 100 can calculate the candidate motion plans thataccount for the possible identifier locations on opposing and/ororthogonal surfaces. Accordingly, the robotic system 100 can account foran opposing pose (e.g., a pose oriented in an opposite direction wherethe outline of the target object stays the same from a viewinglocation/angle) and/or other rotated poses in addition to the initialpose. Referring back to FIG. 3A and FIG. 3C as an illustrative example,the robotic system 100 can calculate the candidate motion plans thataccount for both the first pose 312 and the third pose 316 when thegripping location corresponds to one of the object-peripheral surfaces326.

To account for multiple possible poses (e.g., erroneous estimation ofthe initial pose), the robotic system 100 can calculate a scanning posethat would place the object identifiers in both the exposed identifierset and the alternative identifier set. In some embodiments, asillustrated at block 562, the robotic system 100 can calculate a set ofcandidate poses for the target object 112 in or through the scanningfields. Given the selected approach location, the robotic system 100 cancalculate candidate scanning locations as described above for block 542,such as by rotating and/or translating the identifier location model toplace the identifier location 334 in the scanning field.

At block 564, the robotic system 100 can map the exposed identifier setand the alternative identifier set to each of the candidate scanninglocations. The robotic system 100 can map the exposed identifier setbased on rotating the identifier location model starting from theinitial pose. The robotic system 100 can map the alternative identifierset based on rotating the identifier location model starting from one ofthe alternative poses (e.g., the opposing pose).

With the identifier locations mapped, at block 568, the robotic system100 can compare a location and/or an orientation of the objectidentifiers in both the exposed identifier set and the alternativeidentifier set with the scanning fields. At decision block 570, therobotic system 100 can determine whether the candidate posesimultaneously presents the object identifiers in both the exposedidentifier set and the alternative identifier set to the objectscanners.

At block 572, the robotic system 100 can identify as the scanning posethe candidate poses that simultaneously present the object identifiersin both the exposed identifier set and the alternative identifier set todifferent object scanners/scanning fields. For example, when thegripping location corresponds to one of the object-peripheral surfaces326 with the object locations in the exposed identifier set and thealternative identifier set being on opposing surfaces, the roboticsystem 100 can identify the scanning pose that places the target object112 between a pair of opposing/facing object scanners with each of theopposing surfaces of the target object 112 facing the one of the objectscanners.

At block 574, when none of the candidate poses simultaneously presentsthe object identifiers in both the exposed identifier set and thealternative identifier set, the robotic system 100 can calculatemultiple scanning locations (e.g., a first scanning location and asecond scanning location) that each present at least one objectidentifier from the exposed identifier set and the alternativeidentifier set. For example, the first scanning location can present oneor more object identifier locations in the exposed identifier set to oneof the object scanners, and the second scanning location can present oneor more object identifier locations in the alternative identifier set toone of the object scanners. The second scanning location can beassociated with rotating the end-effector about an axis, translating theend-effector, or a combination thereof from the first scanning location.

Referring back to the example illustrated in FIG. 4, the second motionplan 424 can correspond to the second approach location 434 thatcorresponds to the third surface (e.g., one of the object-peripheralsurfaces 326) that is orthogonal to the two opposing surfaces (e.g., forthe first pose 312 and the third pose 316) as described above.Accordingly, the first scanning location can correspond to a first oneof the second presentation locations 444 that places a surface (e.g.,estimated to be the object-bottom surface 324) corresponding to theinitial pose (e.g., the first pose 312) above and facing anupward-facing object scanner 416. The second scanning location cancorrespond to a second one of the second presentation locations 444 thatrotates the target object 112 90 degrees in a counter-clockwisedirection relative to an overall transfer direction (e.g., generallyfrom the start location 114 to the task location 116). Accordingly, thesecond scanning location can correspond to the second presentationlocation 444 that places a surface (e.g., estimated to be theobject-bottom surface 324) corresponding to the alternative pose (e.g.,the third pose 316) in a vertical orientation in front of and facing ahorizontally facing object scanner 416.

According to the resulting scanning pose and/or the set of scanninglocations, the robotic system 100 can calculate the candidate plans. Therobotic system 100 can use one or more mechanisms described above (e.g.,the A* mechanism) to calculate the candidate plans that place theend-effector at the selected approach location, contact and grip thetarget object 112 accordingly, and lift and transfer the target object112 to the identified scanning pose and/or the set of scanninglocations. For example, when the scanning pose is identified, therobotic system 100 can calculate the candidate plans to establish thescanning pose for the target object 112 in or through the scanningfields. When the robotic system 100 does not identify the scanning pose,the robotic system 100 can calculate the candidate plans totransfer/orient the end-effector sequentially through the set ofmultiple scanning locations, thereby sequentially transferring/rotatingthe target object 112 according to multiple presentationlocations/orientations.

At block 576, the robotic system 100 can recalculate or update thescanning likelihood for each of the candidate motion plans. The roboticsystem 100 can update the scanning likelihood based on combining thevarious probabilities and/or preferences as described above for block544 (e.g., probabilities and/or scores for the approach location, thescanning location, the utilized object scanner, the likely exposedidentifier, an associated error and/or loss rate, or a combinationthereof), but with respect to the scanning metric instead of theperformance metrics.

At block 578, the robotic system 100 can calculate the motion plan basedon selecting the candidate plan according to the scanning likelihood.The robotic system 100 can select the candidate plan that has maximumscanning likelihood among the candidate plans as the motion plan. Forexample, the robotic system 100 can select the candidate plan that hasthe highest likelihood of placing at least one of the exposed identifierlocations and at least one of the alternative identifier locations inone or more of the scanning fields (i.e., before one or more of theobject scanners) during the transfer of the target object 112 (e.g., forscanning in the air between the start location 114 and the task location116).

When more than one candidate plan corresponds to scanning likelihoodswithin a relatively small difference value (e.g., a predeterminedthreshold), the robotic system 100 can calculate and evaluate (e.g., asdescribed above for blocks 544 and 546) performance metricscorresponding to the corresponding candidate plans. The robotic system100 can select the candidate plan that is closest to the targetedcondition as the motion plan.

In some embodiments, the robotic system 100 can deviate from theillustrated example flow. For example, the robotic system 100 can selectthe approach location as described above. Based on the selected approachlocation, the robotic system 100 can grip the target object 112 andimplement a predetermined set of maneuvers, such as to lift, reorient,horizontally translate, place back down and release, or a combinationthereof. During or after the predetermined set of maneuvers, the roboticsystem 100 can re-image or scan the pickup area (via, e.g., looping backto block 502) and redetermine the initial pose and the confidencemeasure (via, e.g., blocks 522 and 524).

Returning back to FIG. 5A, at block 508, the robotic system 100 canbegin implementing the resulting motion plan. The robotic system 100 canimplement the motion plan based on operating the one or more processors202 to send the commands and/or settings of the motion plan to otherdevices (e.g., the corresponding actuation devices 212 and/or otherprocessors) to execute the task 402. Accordingly, the robotic system 100can execute the motion plan by operating the actuation devices 212according to the sequence of commands or settings or combinationthereof. For example, the robotic system 100 can operate the actuationdevices 212 to place the end-effector at the approach location about thestart location 114, contact and grip the target object 112, or acombination thereof.

At block 582, the robotic system 100 can transfer the end-effector tothe scanning location, thereby transferring the target object 112 to thepresentation location/orientation. For example, after or along withlifting the target object 112 from the start location 114, the roboticsystem 100 can transfer the end-effector to establish the scanning posefor the target object 112. Also, the robotic system 100 can transfer theend-effector to the first scanning location.

At block 584, the robotic system 100 can operate the object scanners 416to scan the target object 112. For example, one or more of theprocessors 202 can send a command to the object scanners 416 to performa scan and/or send a query to the object scanners 416 to receive a scanstatus and/or a scanned value. When the motion plan includes thescanning pose, such as at block 585, the robotic system 100 canimplement the motion plan to transfer the target object 112 in thescanning pose across the scanning fields in a direction orthogonal toorientations of the scanning fields. While the target object 112 istransferred, the object scanners 416 can (simultaneously and/orsequentially) scan multiple surfaces for multiple possible locations ofthe object identifier 332.

At decision block 586, the robotic system 100 can evaluate the scanningresult (e.g., status and/or the scanned value) to determine whether thetarget object 112 was scanned. For example, the robotic system 100 canverify the scanning result after implementing the motion plan up to thefirst scanning location. When the scanning result indicates a successfulscan (e.g., the status corresponding to detection of a validcode/identifier and/or the scanned value matching theidentified/expected object) of the target object 112, such as at block588, the robotic system 100 can transfer the target object 112 to thetask location 116. In some embodiments, based on the successful scan,the robotic system 100 can ignore any subsequent scanning location(e.g., the second scanning location) and directly transfer the targetobject 112 to the task location 116.

When the scanning result indicates an unsuccessful scan, the roboticsystem 100 can determine at decision block 590 whether the currentscanning location is the last one in the motion plan. When it is not thelast motion plan, the robotic system 100 can transfer the target object112 to the next presentation location/orientation as represented by aloop back to block 582.

When the current scanning location is the last one in the motion plan,the robotic system 100 can implement one or more remedial actions asillustrated at block 592. In some embodiments, the robotic system 100can halt and/or cancel the motion plan when the scanning results for allof the scanning locations in the motion plan indicate failed scans. Insome embodiments, the robotic system 100 can generate an errorstatus/message for notifying an operator. In some embodiments, therobotic system 100 can place the target object 112 inside of an area(i.e., at a location different from the start location 114 and the tasklocation 116) designated for objects that failed to be scanned.

Based on either successfully completing the task 402 (i.e., successfullyscanning the target object 112 and placing it at the task location 116)or implementing the remedial actions, the robotic system 100 can move onto the next task/object. In some embodiments, the robotic system 100 canrescan the designated areas, as illustrated by a loop back to block 502.In some embodiments, the robotic system 100 can use the existing imagingdata to select the next object as the target object 112, as illustratedby a loop back to block 504.

Scanning the target object 112 in the air (e.g., at a location betweenthe start location 114 and the task location 116) provides improvedefficiency and speed for performing the task 402. By calculating themotion plan that includes the scanning locations and also coordinateswith the object scanners 416, the robotic system 100 can effectivelycombine the task for transferring the target object with the task forscanning the target object. Moreover, calculating the motion planaccording to the confidence measure of the initial orientation furtherimproves efficiency, speed, and accuracy for the scanning task. Asdescribed above, the robotic system 100 can calculate the motion planthat accounts for alternative orientations that correspond to thescenario of the initial pose being inaccurate. Accordingly, the roboticsystem 100 can increase the likelihood of accurately/successfullyscanning the target object even with pose determination errors, such asdue to calibration errors, unexpected poses, unexpected lightingconditions, etc. The increased likelihood in accurate scans can lead toincreased overall throughput for the robotic system 100 and furtherreduce operator efforts/interventions.

CONCLUSION

The above Detailed Description of examples of the present disclosure isnot intended to be exhaustive or to limit the present disclosure to theprecise form disclosed above. While specific examples for the presentdisclosure are described above for illustrative purposes, variousequivalent modifications are possible within the scope of the presentdisclosure, as those skilled in the relevant art will recognize. Forexample, while processes or blocks are presented in a given order,alternative implementations may perform routines having steps, or employsystems having blocks, in a different order, and some processes orblocks may be deleted, moved, added, subdivided, combined, and/ormodified to provide alternative or sub-combinations. Each of theseprocesses or blocks may be implemented in a variety of different ways.Also, while processes or blocks are at times shown as being performed inseries, these processes or blocks may instead be performed orimplemented in parallel, or may be performed at different times.Further, any specific numbers noted herein are only examples;alternative implementations may employ differing values or ranges.

These and other changes can be made to the present disclosure in lightof the above Detailed Description. While the Detailed Descriptiondescribes certain examples of the present disclosure as well as the bestmode contemplated, the present disclosure can be practiced in many ways,no matter how detailed the above description appears in text. Details ofthe system may vary considerably in its specific implementation, whilestill being encompassed by the present disclosure. As noted above,particular terminology used when describing certain features or aspectsof the present disclosure should not be taken to imply that theterminology is being redefined herein to be restricted to any specificcharacteristics, features, or aspects of the present disclosure withwhich that terminology is associated. Accordingly, the invention is notlimited, except as by the appended claims. In general, the terms used inthe following claims should not be construed to limit the presentdisclosure to the specific examples disclosed in the specification,unless the above Detailed Description section explicitly defines suchterms.

Although certain aspects of the invention are presented below in certainclaim forms, the applicant contemplates the various aspects of theinvention in any number of claim forms. Accordingly, the applicantreserves the right to pursue additional claims after filing thisapplication to pursue such additional claim forms, in either thisapplication or in a continuing application.

I claim:
 1. A method for operating a robotic system, the methodcomprising: receiving imaging data representing a pickup area includinga target object; determining an initial pose of the target object basedon the imaging data; calculating a confidence measure associated withthe initial pose, wherein the confidence measure is indicative of alikelihood of the initial pose being accurate; deriving, based on theconfidence measure, an approach location and at least one scanninglocation, the approach location defining a location for an end-effectorof a robotic arm to contact and grip the target object, the scanninglocation defining a location for the end-effector to present one or moresurfaces of the target object before one or more corresponding objectscanners so as to scan one or more object identifiers; deriving, basedon the derived approached location and the scanning location, a motionplan, wherein the motion plan includes commands, settings, or acombination thereof, for operating the robotic arm and the end-effectorto execute a task based on (1) picking up the target object from a startlocation, (2) transferring the target object to a task location, and (3)presenting the one or more object identifiers to the one or more objectscanners located between the start location and the task location; andimplementing the motion plan for operating one or more actuation devicesto execute the task, wherein implementing the motion plan includes:determining a scanning result; halting the motion plan when the scanningresult indicates a failed scan; and; implementing a failure resolutionfor notifying an operator, for placing the target object at a locationdifferent from the start location and the task location, or acombination thereof; wherein deriving the approach location and the atleast one scanning location and deriving the motion plan include:comparing the confidence measure to a sufficiency threshold, selectivelycalculating the approach location and the scanning location according toa performance metric and/or a scanning metric, based on an outcome ofthe comparison, wherein the scanning metric corresponds to a likelihoodof at least one of the object identifiers remaining uncovered by theend-effector regardless of whether or not the initial pose is accurate.2. The method of claim 1, wherein deriving the approach location and thescanning location includes deriving the approach location and thescanning location according to the scanning metric when the confidencemeasure fails to satisfy the sufficiency threshold.
 3. The method ofclaim 1, wherein deriving the approach location and the scanninglocation includes prioritizing the scanning metric before theperformance metric when the confidence measure fails to satisfy thesufficiency threshold.
 4. The method of claim 1, wherein deriving theapproach location and the scanning location includes deriving theapproach location and the scanning location based on the performancemetric when the confidence measure satisfies the sufficiency threshold.5. The method of claim 1, wherein deriving the approach location and thescanning location includes: calculating a set of available approachlocations about the start location for the target object, wherein theset of available approach locations for placing the end-effector tocontact and grip the target object without disturbing other objects;calculating an exposed identifier set for each available approachlocation in the set of available approach locations, wherein the exposedidentifier set represents corresponding locations of the objectidentifiers remaining uncovered with the end-effector at thecorresponding approach location according to a first hypothesis of theinitial pose being accurate; calculating an alternative identifier setfor each available approach location in the set of available approachlocations, wherein the alternative identifier set represents locationsof the object identifiers remaining uncovered with the end-effector atthe corresponding approach location according to a second hypothesis ofthe initial pose being inaccurate; and selecting the approach location,wherein the approach location is the available approach location havinguncovered object identifiers in both the exposed identifier set and thealternative identifier set.
 6. The method of claim 5, furthercomprising: identifying scanning fields corresponding to the objectscanners; calculating a set of candidate poses for the target object inor through the scanning fields, wherein each candidate pose presentsmultiple surfaces of the target object to the object scanners; mappingthe exposed identifier set and the alternative identifier set to each ofthe candidate poses; comparing the exposed identifier set and thealternative identifier set for each candidate pose to the scanningfields; identifying a scanning pose as a candidate pose presenting theobject identifiers in the exposed identifier set and the alternativeidentifier set to the scanning fields; and wherein deriving the motionplan includes: deriving the motion plan establishing the scanning posein or through the scanning fields.
 7. The method of claim 6, wherein:identifying scanning fields includes identifying scanning fieldsoriented in opposite directions, orthogonal directions, or a combinationthereof for scanning different surfaces of the target object in onepose; and implementing the motion plan includes implementing the motionplan for transferring the target object in the scanning pose across thescanning fields in a direction orthogonal to orientations of thescanning fields.
 8. The method of claim 5, wherein deriving the motionplan includes calculating at least a first scanning location and asecond scanning location, wherein the first scanning location is forpresenting a location in the exposed identifier set to one of the objectscanners and the second scanning location is for presenting a locationin the alternative identifier set to one of the object scanners.
 9. Themethod of claim 8, wherein implementing the motion plan includes:verifying a scanning result after implementing the motion plan up to thefirst scanning location; and transferring the target object to the tasklocation and ignoring the second scanning location when the scanningresult represents a successful scan.
 10. The method of claim 8, whereinimplementing the motion plan includes: verifying a scanning result afterimplementing the motion plan up to the first scanning location; andtransferring the target object to the second scanning location when thescanning result represents a failed scan.
 11. The method of claim 5,further comprising: comparing the exposed identifier set to thealternative identifier set to determine whether the exposed identifierset and the alternative identifier set include locations on opposingsurfaces of the target object; and selecting the approach locationassociated with a third surface orthogonal to the opposing surfaces. 12.The method of claim 11, wherein deriving the approach location and thescanning location includes deriving the approach location and thescanning location without considering the scanning metric when theconfidence measure satisfies the sufficiency threshold.
 13. The methodof claim 1, wherein deriving the approach location and the at least onescanning location and deriving the motion plan includes: calculating aset of available approach locations about the start location for thetarget object, wherein the set of available approach locationsrepresents positions for placing the end-effector to grip the targetobject without disturbing other objects; calculating a candidate planfor each available approach location in the set of available approachlocations, wherein the candidate plan includes a candidate scanninglocation; calculating the performance metric for each candidate plan,wherein the performance metric corresponds to a throughput rateassociated with a distance traversed by the target object, an estimatedtransfer duration, a number of commands or setting changes, a completionrate associated with the corresponding available approach location, or acombination thereof for placing the target object at the task location;and selecting the approach location, wherein the approach location is anavailable approach location in the set corresponding to the highestperformance metric.
 14. The method of claim 1, wherein determining theinitial pose of the target object includes: identifying a set of edgesof an exposed surface of the target object in the imaging data;estimating physical dimensions of the target object based on measuringthe edges in the set and mapping the measurements into standard lengths;and determining the initial pose based on identifying the exposedsurface, including matching the estimated physical dimensions of theexposed surface to a set of known dimensions of object surfaces inmaster data.
 15. The method of claim 14, wherein: determining theinitial pose includes identifying a pair of opposing surfaces matchingthe estimated physical dimensions; and when the confidence measure failsto satisfy the sufficiency threshold, calculating the motion plan thataccounts for an opposing pose in addition to the initial pose, whereinthe opposing pose and the initial pose correspond to the pair ofopposing surfaces.
 16. The method of claim 14, wherein determining theinitial pose includes determining the initial pose based on identifyingan exposed surface, an orientation thereof, or a combination thereofusing a visible mark on the exposed surface, including matching thevisible mark to a set of known marks on object surfaces in master data.17. A tangible, non transitory computer-readable medium having processorinstructions stored thereon that, when executed by a robotic system,cause the robotic system to perform a method comprising: receivingimaging data representing a pickup area including a target object;determining an initial pose of the target object based on the imagingdata; calculating a confidence measure associated with the initial pose,wherein the confidence measure is indicative of a likelihood of theinitial pose being accurate; deriving, based on the confidence measure,an approach location and at least one scanning location, the approachlocation defining a location for an end-effector of a robotic arm tocontact and grip the target object, the scanning location defining alocation for the end-effector to present one or more surfaces of thetarget object before one or more corresponding object scanners so as toscan the one or more object identifiers, deriving, based on the derivedapproached location and the scanning location, a motion plan, whereinthe motion plan includes commands, settings, or a combination thereof,for operating the robotic arm and the end-effector to execute a taskbased on (1) picking up the target object from a start location, (2)transferring the target object to a task location, and (3) presentingthe one or more object identifiers to the one or more object scannerslocated between the start location and the task location, andimplementing the motion plan for operating one or more actuation devicesto execute the task, wherein implementing the motion plan includes:determining a scanning result; halting the motion plan when the scanningresult indicates a failed scan; and implementing a failure resolutionfor notifying an operator, for placing the target object at a locationdifferent from the start location and the task location, or acombination thereof; wherein deriving the approach location and the atleast one scanning location and deriving the motion plan include:comparing the confidence measure to a sufficiency threshold, andselectively calculating the approach location and the scanning locationaccording to a performance metric and/or a scanning metric, based on anoutcome of the comparison, wherein the scanning metric corresponding toa likelihood of at least one of the object identifiers remaininguncovered by the end-effector regardless of whether or not the initialpose is accurate.
 18. The tangible, non transitory computer-readablemedium of claim 17, wherein deriving the approach location and thescanning location: includes deriving the approach location and thescanning location according to the scanning metric when the confidencemeasure fails to satisfy the sufficiency threshold.
 19. A robotic systemcomprising: at least one processor; and at least one memory deviceoperably coupled to the processor and having stored thereon instructionsexecutable by the processor to: receive imaging data representing apickup area including a target object; determine an initial pose of thetarget object based on the imaging data; calculate a confidence measureassociated with the initial pose, wherein the confidence measure isindicative of a likelihood of the initial pose being accurate; derive,based on the confidence measure, an approach location and at least onescanning location, the approach location defining a location for anend-effector of a robotic arm to contact and grip the target object, thescanning location defining a location for the end-effector to presentone or more surfaces of the target object before one or morecorresponding object scanners so as to scan one or more objectidentifiers; derive, based on the derived approached location and thescanning location, a motion plan, wherein the motion plan includescommands, settings, or a combination thereof, for operating the roboticarm and the end-effector to execute a task based on (1) picking up thetarget object from a start location, (2) transferring the target objectto a task location, and (3) presenting the one or more objectidentifiers to the one or more object scanners located between the startlocation and the task location; and implement the motion plan foroperating one or more actuation devices to execute the task, whereinimplementing the motion plan includes: determining a scanning result;halting the motion plan when the scanning result indicates a failedscan; and implementing a failure resolution for notifying an operator,for placing the target object at a location different from the startlocation and the task location, or a combination thereof; whereinderiving the approach location and the at least one scanning locationand deriving the motion plan include: comparing the confidence measureto a sufficiency threshold; and selectively calculating the approachlocation and the scanning location according to a performance metricand/or a scanning metric, based on an outcome of the comparison, whereinthe scanning metric corresponds to a likelihood of at least one of theobject identifiers remaining uncovered by the end-effector regardless ofwhether or not the initial pose is accurate.
 20. The robotic system ofclaim 19, wherein the memory device includes instructions executable bythe processor to: calculate a set of available approach locations aboutthe start location for the target object, wherein the set of availableapproach locations for placing the end-effector to contact and grip thetarget object without disturbing other objects; calculate an exposedidentifier set for each available approach location in the set ofavailable approach locations, wherein the exposed identifier setrepresents corresponding locations of the object identifiers remaininguncovered with the end-effector at the corresponding approach locationaccording to a first hypothesis of the initial pose being accurate;calculate an alternative identifier set for each available approachlocation in the set of available approach locations, wherein thealternative identifier set represents locations of the objectidentifiers remaining uncovered with the end-effector at thecorresponding approach location according to a second hypothesis of theinitial pose being inaccurate; and select as the approach location theavailable approach location having uncovered object identifiers in boththe exposed identifier set and the alternative identifier set.